Add filtering to fieldcaps endpoint #83636

romseygeek · 2022-02-08T09:57:25Z

Many consumers of the field caps API need to do some post-processing of the
results before they can use them; for instance, Kibana would like to exclude
multifields from certain field selections, or would like to display only geo_point
fields in Maps. ML and QL consumers exclude nested fields in certain
circumstances. This post-processing is possible at the moment, but can be
hacky; and in all cases it involves sending the whole (possibly very large) field
caps response over the wire and then whittling it down in the client. It is also not
guaranteed to be accurate - runtime fields may be incorrectly classified as multifields,
for example.

This commit pushes filtering into elasticsearch itself, reducing the amount of data
that needs to be transported and ensuring better accuracy. The field caps API gets
two new parameters:

filters - a comma-delimited list that may contain any combination of: +metadata,
-metadata, -nested, -parent, -multifield
types - a comma-delimited list of field types; only fields that have a type in this set
will be returned

The API will make best-effort attempts to apply the filters post-hoc to responses from
older nodes, so this should still work in a mixed-cluster or cross-cluster situation.

Fixes #82966, #72174

elasticmachine · 2022-02-08T09:57:28Z

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine · 2022-02-08T09:57:50Z

Hi @romseygeek, I've created a changelog YAML for you.

…eldcaps/filtering

elasticmachine · 2022-02-08T13:13:06Z

Pinging @elastic/clients-team (Team:Clients)

sethmlarson · 2022-02-08T15:04:31Z

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/field_caps/50_fieldtype_filter.yml

+  - is_true: fields.misc\\.keyword
+
+---
+"Field type filters":


Can we add a test case for multiple values in the fields parameter?

javanna

I started reviewing this a while ago, then I got sidetracked, but I eventually got back to it. I left some comments that I think we should address, even if this PR has already been merged.

javanna · 2022-02-22T15:26:40Z

docs/reference/search/field-caps.asciidoc

@@ -77,6 +77,15 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailab
  (Optional, Boolean) If `true`, unmapped fields are included in the response.
  Defaults to `false`.

+`filters`::
+(Optional, string) Comma-separated list of filters to apply to the response.  The
+following filters are supported: +metadata,-metadata,-parent,-nested,-multifield


why is the + notation supported only for metadata? Would it make sense to expand on what each one of these mean, with a link to the relevant docs explanation for each one?

I'll expand the docs. The idea is that - and + work in much the same way they do in our query parsers = - means 'must not match', and + means 'must match'. So you can ask for only metadata, or to exclude metadata. For the others it doesn't make much sense to do anything except exclude them, with the exception of parent where it sort of makes sense to say 'include parent data as well as everything else', so a bare parent term is supported.

server/src/main/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesFetcher.java

javanna · 2022-03-07T08:47:20Z

docs/reference/search/field-caps.asciidoc

+following filters are supported: +metadata,-metadata,-parent,-nested,-multifield
+
+`types`::
+(Optional, string) Comma-separated list of field types to include. Any fields that


shall we link the docs page with all of the field types here? Maybe also include an example at the end of the sentence?

javanna · 2022-03-07T08:56:38Z

docs/reference/search/field-caps.asciidoc

+`types`::
+(Optional, string) Comma-separated list of field types to include. Any fields that
+do not match one of these types will be excluded from the results.  Defaults to empty,
+meaning that all field types are returned.


One more question, just to double check: types:keyword means either runtime or indexed field, no distinction between the two for field_caps?

Keyword runtime fields return KeywordFieldMapper.CONTENT_TYPE as their family name (which is what is actually checked here). I also wonder if it's worth adding a -runtime, +runtime pair of filters. I know you don't like making it too obvious what is runtime and what isn't, but I think it can still be helpful for clients.

let's wait for them to ask? :)

javanna · 2022-03-07T08:57:50Z

server/src/main/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesFetcher.java

+            if ("-parent".equals(filter)) {
+                return false;
+            }
+            if ("parent".equals(filter)) {


I don't see this one documented: is it +parent or just parent?

This is just plain parent and acts as the equivalent of a SHOULD clause - in other words, do return parent information. Clients that aren't filtering out nested or multifield entries still use this to do their own checks, but once they start using filters they can filter out parents instead. At some point in the future I envisage that we will filter out parents by default, and then clients will have to explicitly include parent if they still want that information.

server/src/main/java/org/elasticsearch/action/fieldcaps/ResponseRewriter.java

javanna · 2022-03-07T09:35:15Z

server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java

@@ -336,6 +336,9 @@ public NestedLookup nestedLookup() {
    }

    public boolean isMultiField(String field) {
+        if (fieldTypeLookup.isRuntimeField(field)) {
+            return false;
+        }


I see that there are existing callers of this method in FieldMapper. Was it not accurate before? I get thrown off that we look at field mappers later, and runtime fields don't have field mappers, so why do we need this additional if and what effect does it have?

This is used when validating copy_to fields, in which case we don't really care about runtime fields. But it is important for field cap filtering, because otherwise we might incorrectly a runtime field foo.bar as a multifield of an otherwise existing field foo.

Thinking about it though, I reckon we can do this without looking at runtime fields just by checking to see if both the passed-in field name and its parent are field mappers. Let me see what I can come up with.

javanna · 2022-03-07T09:36:34Z

server/src/test/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesNodeRequestTests.java

+                    instance.runtimeFields()
+                );
+            }
+            default -> throw new IllegalStateException("The test should only allow 7 parameters mutated");


I think we are missing a serialization tests that uses multiple versions, to exercise the bwc layer.

javanna · 2022-03-07T09:36:45Z

server/src/test/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesRequestTests.java

            )
        );
        assertThat(
            request.getDescription().length(),
-            lessThanOrEqualTo(1024 + ("indices[" + "s9999,... (9999 in total, 9999 omitted)], fields[a]").length())
+            lessThanOrEqualTo(1024 + ("indices[" + "s9999,... (9999 in total, 9999 omitted)], fields[a], filters[], types[]").length())


here too, I think we are missing a serialization tests that uses multiple versions, to exercise the bwc layer.

server/src/main/java/org/elasticsearch/index/mapper/FieldTypeLookup.java

romseygeek · 2022-03-07T12:40:18Z

Thanks for all the comments @javanna, I will open a follow-up PR to address them.

* adds a test for mixed cluster requests * fixes a bad stream version check (above test will fail if this isn't included) * replaces private FieldCapsFilter interface with Predicate * renames 'allowedTypes' to 'types' to maintain consistency with external API * adds javadoc to ResponseRewriter * removes isRuntimeField from FieldTypeLookup Relates to #83636

romseygeek added 8 commits January 28, 2022 09:46

Add filters to field caps + bwc

abe01d8

WIP

3771d63

Merge remote-tracking branch 'origin/master' into fieldcaps/filtering

27687fe

nuke timeseries filter as it's not useful for now

5a6868e

add yaml tests

86f932e

Add filtering on field type

2679222

Docs

32a5c35

Merge remote-tracking branch 'origin/master' into fieldcaps/filtering

1a65377

romseygeek added >enhancement :Search/Search Search-related issues that do not fall into other categories v8.2.0 labels Feb 8, 2022

romseygeek requested review from nik9000, javanna and thomasneirynck February 8, 2022 09:57

romseygeek self-assigned this Feb 8, 2022

elasticmachine added the Team:Search Meta label for search team label Feb 8, 2022

romseygeek and others added 4 commits February 8, 2022 09:57

Update docs/changelog/83636.yaml

851755f

spotless

6c0d3e9

rename

196016f

Merge remote-tracking branch 'romseygeek/fieldcaps/filtering' into fi…

37a70fc

…eldcaps/filtering

sethmlarson added the Team:Clients Meta label for clients team label Feb 8, 2022

Merge remote-tracking branch 'origin/master' into fieldcaps/filtering

31da51b

sethmlarson reviewed Feb 8, 2022

View reviewed changes

romseygeek added 2 commits February 9, 2022 11:43

Merge remote-tracking branch 'origin/master' into fieldcaps/filtering

375ad2c

Add test including fields restriction

2a9f5d7

javanna removed the request for review from nik9000 February 9, 2022 13:40

Merge remote-tracking branch 'origin/master' into fieldcaps/filtering

77d0557

Extend tests to use a fields list

7d65530

sethmlarson approved these changes Feb 10, 2022

View reviewed changes

romseygeek merged commit 8bc46ad into elastic:master Feb 10, 2022

romseygeek deleted the fieldcaps/filtering branch February 10, 2022 14:06

romseygeek mentioned this pull request Feb 28, 2022

Add ability to find fields by field-type #72174

Closed

nreese mentioned this pull request Feb 28, 2022

[maps] use field_caps filtering to find data views with geo_point and geo_shape fields elastic/kibana#126478

Open

javanna reviewed Mar 7, 2022

View reviewed changes

romseygeek mentioned this pull request Mar 7, 2022

Expose information about multi-fields in _field_caps #75474

Open

javanna mentioned this pull request Mar 21, 2022

Extra testing and some cleanups for filtering on field caps #85068

Merged

javanna mentioned this pull request Jun 16, 2022

Multi-fields information in field caps API #58357

Closed

nreese mentioned this pull request Feb 13, 2023

Data views usage collection causes large amount of Elasticsearch queries elastic/kibana#151064

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add filtering to fieldcaps endpoint #83636

Add filtering to fieldcaps endpoint #83636

romseygeek commented Feb 8, 2022

elasticmachine commented Feb 8, 2022

elasticsearchmachine commented Feb 8, 2022

elasticmachine commented Feb 8, 2022

sethmlarson Feb 8, 2022

javanna left a comment

javanna Feb 22, 2022

romseygeek Mar 7, 2022

javanna Mar 7, 2022

javanna Mar 7, 2022

romseygeek Mar 7, 2022

javanna Mar 7, 2022

javanna Mar 7, 2022

romseygeek Mar 7, 2022

javanna Mar 7, 2022

romseygeek Mar 7, 2022

javanna Mar 7, 2022

javanna Mar 7, 2022

romseygeek commented Mar 7, 2022

Add filtering to fieldcaps endpoint #83636

Add filtering to fieldcaps endpoint #83636

Conversation

romseygeek commented Feb 8, 2022

elasticmachine commented Feb 8, 2022

elasticsearchmachine commented Feb 8, 2022

elasticmachine commented Feb 8, 2022

Choose a reason for hiding this comment

javanna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

romseygeek commented Mar 7, 2022